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Abstract 

The  UK  Ministry  of  Defence  (MoD)  spends  £2.6  billion  per  year  (MoD,  2008,  Section 
1.2;  2007,  p.  2)  on  research  and  development  (R&D)  (“Defense  Technology,”  2009,  p.  22). 
The  figure  is  technically  correct,  but  it  conceals  more  than  it  reveals.  Of  the  total,  the  MoD 
spends  around  £500  million  (MoD,  2006,  p.  8)  on  laboratory  research  and  on  taking  what 
emerges  from  the  lab  in  its  first  few  steps  down  the  long  road  that  leads  eventually  to  mature 
technology  embedded  in  military  equipment.  Analysis  based  on  UK  National  Audit  Office 
(NAO)  data  (Stationery  Office,  2006,  November  24;  2008,  p.  5)  shows  that  about  half  of 
project  timescale  overruns  are  due  to  technology  maturation  occurring  too  late  (Jordan  & 
Dowdy,  2007,  p.  16).  US  evidence  shows  that  defence  projects  that  get  all  their  technology 
mature  before  the  equivalent  of  Main  Gate  suffer  only  very  small  time  and  cost  overruns 
thereafter  (GAO,  2007,  pp.  14-15). 
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The  first  hypothesis  would  be  that  funding  for  technology  development  occurs  too 
late  in  the  acquisition  process,  when  the  problems  that  inevitably  occur  have  a 
disproportionate  effect  on  project  timescales  and  costs.  However,  is  this  construct  based  on 
project  failings  or  funding  process  failures?  To  improve  the  outcome,  could  we  simply 
improve  the  timing  allocation  of  funds?  This  research  examines  the  profile  of  funding  as 
aligned  to  maturity  levels  of  technology  and  system  and  integration  readiness,  and  makes 
proposals  on  the  improvements  that  could  be  made. 

Keywords:  Finance,  acquisition,  maturity,  maturity  levels,  timescales,  system 
readiness,  audit,  MoD;  ministry  of  defence;  defence,  defense,  systems  engineering,  Dstl; 
Defence  Science  and  Technology  Laboratory,  National  Audit  Office,  NAO. 

When  is  the  Best  Time  to  Spend — Globally? 

Organisations  that  procure  products  or  services  through  contracts  are  routinely 
involved  in  costing  and  estimation  efforts.  Defence  Acquisition  is  a  long-term  activity,  and 
the  success  or  failure  of  the  endeavor  can,  in  part,  be  related  to  the  stability  of  the  aims  and 
contributory  components. 

Economic  growth  has  been  driven  by  globalisation  over  the  last  30  or  more  years, 
generating  networks  of  connections  and  interdependencies  between  the  major  economic 
powers  that  are  unprecedented  in  extent  and  pervasiveness  (DCDC,  2009).  The  economic 
landscape  has  evolved  rapidly  with  the  demise  of  centrally  planned  economies,  such  as  the 
Soviet  Union;  the  rise  of  Asian  economies,  particularly  China,  which  has  embraced  a  market 
aware  philosophy;  and  the  maturation  of  the  European  Union  as  a  cohesive  economic 
market.  These  changes  have  created  a  multi-faceted  economic  landscape  that  is  intimately 
interconnected  and  influential. 

Over  the  last  30  years,  the  global  economy  has  grown  at  a  trend  rate1  of  3-4%  (IMF, 
2008),  and  output  has  seen  a  greater-than  four-fold  increase.  Economic  growth,  combined 
with  the  continuing  rise  in  the  global  population,  will  intensify  the  demand  for  natural 
resources,  minerals,  and  energy.  It  will  continue  to  be  uneven,  fluctuating  over  time  and 
between  regions  (Goldman  Sachs,  2007).  This  global  economic  context  places  more 
pressure  on  all  dimensions  of  UK  government  funding,  and  with  the  economic  recession 
being  felt  in  the  homeland  environment,  defence  focus  shifts  from  operational  reach  to 
border  security.  Operational  reach  is  the  defensive  objective  of  nations  in  times  of  affluence; 
border  security  is  the  focus  during  recession. 

The  IMF  considers  there  have  been  five  global  recessions  in  the  last  30  years  (IMF, 
2008).  Global  economic  recessions  will  happen  over  the  next  few  decades,  and 
governments  are  likely  to  respond  to  them  with  protectionist  policies,  designed  to  shield  their 
own  economies  and  workforces  from  global  competition.  The  global  recession  that  started  in 
2008  is  illustrative  of  the  likely  response  to,  and  effects  of,  future  recessions.  One  outcome 
of  a  global  recession  is  an  expected  increase  in  the  incidence  of  poverty,  which  will  promote 
grievance  and  dissatisfaction  among  those  who  suffer  economic  hardship,  in  turn  breeding 
political  violence,  criminality,  societal  conflict  and  the  destabilization  of  states  and  regions 
unable  to  cope.  This  type  of  economic  crisis  will  produce  pressures  on  government  finances 
that,  considered  in  isolation,  will  place  downward  pressure  on  global  defence  spending. 


1  Trend  economic  growth  refers  to  average  growth  of  an  economy  over  a  cycle  of  expansion  and  contraction.  It  is  a  moving  average.  Global  recession  is  defined  as 
annual  world  growth  (based  on  purchasing-power-parity  weights)  falling  below  3%.  By  that  measure,  four  periods  since  1980  qualify:  1980-1984,  1990-1993,  1998,  2001- 
2002,  plus  2008  onwards.  The  figure  of  3%  is  a  rough  correlation  to  negative  global  GDP  per  capita  growth  (IMF,  2008). 
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In  September  2009  (“Britain’s  Sharp,”  2009),  the  UK  Chancellor  of  the  Exchequer 
forecasted  that  in  April  2010  public  borrowing  would  exceed  12%  of  GDP.  This  projection 
would  give  Britain  the  biggest  debt  build-up  between  2007  and  2014  in  the  G7  economies. 
Gaining  credibility  will  require  clear  targets  to  cut  the  deficit,  together  with  detailed  and 
transparent  plans  on  how  to  achieve  them.  A  crucial  decision  will  be  the  mix  of  spending 
cuts  and  tax  rises.  In  the  UK,  public  spending  represents  close  to  50%  of  GDP,  according  to 
the  Treasury.  In  the  March  2010  UK  Budget,  the  plan  proposed  was  to  halve  the  level  of 
borrowing,  to  5.2&  of  national  income,  by  2013-14  (“Economic,”  2010).  The  chancellor 
stated  that  spending  plans  from  201 1  will  be  “very  tough — the  toughest  for  decades,"  but 
refused  to  outline  how  this  will  affect  public  services,  except  to  say  that  the  Government 
would  “protect  spending  on  those  frontline  public  services  on  which  we  all  depend .”  The 
annual  funding  review  effects  all  government  departments  and  has  inter-related  impacts  on 
the  services  and  support  that  can  be  offered.  In  the  US,  the  current  focus  has  been  on  the 
US  health  systems  and  associated  reforms.  In  the  UK,  balancing  the  books  and  the 
impending  election  process  has  raised  the  level  of  scrutiny  and  resulted  in  long-term  plans 
being  put  on  hold  until  the  political  direction  and  stability  have  been  secured  (Arran,  Gale  & 
Orszag,  2004). 

In  March  2010,  UK  expenditure  per  government  department  can  be  represented  as  a 
pie  chart  showing  the  key  funded  government  departments  (Figure  1)  (“HM  Treasury,” 

2010).  The  allocation  of  spending  to  functions  is  largely  based  on  the  United  Nations’ 
Classification  of  the  Functions  of  Government  (COFOG)  guidelines.  Other  expenditures 
include  general  public  services,  including  international  services,  recreation,  culture  and 
religion,  public  service  pensions,  plus  spending  yet  to  be  allocated  and  some  accounting 
adjustments.  The  Defence  spending  is  £40  billion,  which  represents  5.6%  of  the  £704  billion 
total. 
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Looking  at  the  global  and  national  data,  it  could  be  argued  that  “no”  it  is  not  the  time 
to  spend.  Any  expenditure  commitments  at  this  time  will  disappear  into  the  deficit  arrears, 
and  any  actual  gains  will  be  lost.  However,  funding  streams  need  to  be  spent  to  ensure  that 
the  position  does  not  regress,  and  so  investment  is  required  to  ensure  the  maintenance  of 
current  position  and  level  of  capability  and  service. 

In  the  current  global  and  national  climate,  it  is  not  a  good  time  to  spend.  However,  to 
maintain  capability  in  defence  and  other  sectors,  spending  has  to  be  sustained  to  ensure 
stability  and  maintenance.  In  terms  of  defence  projects,  some  projects  are  pending 
approval,  some  projects  are  committed  by  still  under  development  and  some  projects  are 
further  down  the  in-service  and  support  stage.  For  committed  projects,  the  global  and 
national  situation  can  be  considered  as  relevant,  but  not  a  significant  issue.  The  national 
situation  will  determine  the  size  of  the  overall  budget,  but  how  and  when  it  is  spent  is  an 
issue  for  the  project  team.  Systems  engineers  often  serve  as  technical  points  of  contact 
throughout  the  entire  system  lifecycle  (Smartt  &  Ferreira,  2010);  as  such,  they  are  one  of  the 
critical  repositories  of  project  life  knowledge,  technically  and  financially  (Valerdi,  Rieff, 
Roedler,  Wheaton  &  Wang,  2007).  Using  the  natural  holistic  skills  of  a  systems  engineer, 
and  applying  those  inherent  skills  to  costing,  has  been  the  basis  of  this  research. 

To  date,  there  has  been  significant  research  into  through  life  costing  (Haskins, 
Forsberg  &  Krueger,  2007;  Valerdi  &  Miller,  2007), 2  and  the  projection  of  through  life 
implications  is  based  on  existing  data,  extrapolated  to  a  conclusion.  This  method  is  well 
documented  and  appealing,  as  it  reinforces  the  view  that  we  can  corporately  learn  from 
experience  and  that  costing  is  achieved  within  a  defined  set  of  rules.  It,  therefore,  follows 
that  if  we  can  define  the  rules,  then  we  can  define  the  outcome  (Stationery  Office,  2007; 
2008).  This  research  examines  the  project  costing  profile,  as  related  to  three  measures  of 
readiness  that  are  well  defined  in  the  systems  domain.  These  three  were  selected  because 
they  represent  a  global  viewpoint  and  are  used  in  a  number  of  multinational  and  multilingual 
programmes. 

Optimum  Technology,  System  and  Integration  Investment  Date 

Technology  assessment  of  a  complex  system  requires  the  assessment  of  all  of  its 
systems,  sub-systems  and  components.  Technology  Readiness  Levels  (TRLs)  can  be  used 
to  determine  the  current  component  maturity.  Although  broadly  similar,  different  international 
agencies  use  different  definitions  for  TRLs;  the  most  common  definitions  are  those  used  by 
the  Department  of  Defense  (DoD)  (DoD,  2006)  and  the  National  Aeronautics  and  Space 
Administration  (NASA)  (Mankins,  1995)  (Table  1  and  Figure  2).  In  Europe,  the  European 
Space  Agency’s  (ESA)  definitions  of  TRLs  are  often  used  as  contractual  baseline  criteria  for 
project  acceptance  (ESA?  n.d.).  A  TRL  calculator  has  been  developed  for  the  United  States  Air 
Force  by  Nolte  et  al.  (2003).  As  with  all  assessments,  this  TRL  calculator  provides  a 
snapshot  of  technology  maturity  at  a  given  point  in  time.  The  Technology  Program 
Management  Model  (TPMM)  was  developed  by  the  United  States  Army  by  Craver  et  al. 
(2006).  The  TPMM  is  a  TRL-gated  high-fidelity  activity  model  that  can  be  used  as  a 
management  tool  to  assist  in  planning,  managing,  and  assessing  technologies  for 
successful  technology  transition.  The  model  provides  a  core  set  of  activities,  including 
systems  engineering  and  program  management  tasks,  that  are  tailored  to  the  technology 
development  and  management  goals. 


2  Through  Life  Costing  is  a  defined  UK  Ministry  of  Defence  (MoD)  process  ( Acquisition ,  2009). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


687 


Table  1 .  Table  of  TRL  Descriptions 


National  Aeronautics  and  Space  Administration  (NASA)  Technology 
Readiness  Level 

1.  Basic  principles  observed  and  reported 


2.  Technology  concept  and/or  application  formulated 

3.  Analytical  and  experimental  critical  function  and/or  characteristic  proof 
of  concept 

4.  Component  and/or  breadboard  validation  in  laboratory  environment 


5.  Component  and/or  breadboard  validation  in  relevant  environment 

6.  System/subsystem  model  or  prototype  demonstration  in  a  relevant 
environment  (ground  or  space) 

7.  System  prototype  demonstration  in  a  space  environment 

8.  Actual  system  completed  and  'flight  qualified'  through  test  and 
demonstration  (ground  or  space) 

9.  Actual  system  'flight  proven'  through  successful  mission  operations 


System  Test,  Launch 
&  Operations 


System/Subsystem 

Development 


Technology 

Demonstration 


Technology 

Development 


Research  to  Prove 
Feasibility 


Basic  Technology 
Research 


Figure  1.  NASA  TRL  Diagram 

System  Readiness  Levels  (SRLs)  (“What  Are,”  2009)  have  been  developed  as  a 
project  management  tool  to  capture  evidence,  and  assess  and  communicate  system 
maturity  in  a  consistent  manner  to  stakeholders.  In  the  UK,  SRLs  define  a  set  of  nine 
maturity  steps  from  concept  to  in-service  across  a  set  of  systems  engineering  disciplines. 
Each  of  the  SRL  steps  align  to  key  outputs  from  systems  disciplines  such  as  Training, 

Safety  and  Environmental,  or  Reliability  and  Maintainability.  SRLs  aim  to  take  a 
consolidated  view  of  the  essential  steps  needed  to  properly  mature  and  deliver  a  complete 
supportable  system  to  the  end  user.  SRLs  are  a  means  of  analysing  key  outputs  of  an 
equipment  acquisition  project,  structured  in  such  a  way  as  to  provide  an  understanding  of 
work  required  to  mature  the  project.  In  the  UK,  a  matrix  is  used  to  capture  these  key  outputs 
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and  understand  how  they  should  mature  over  time.  Projects  track  their  maturity  through  the 
nine  SRL  steps  across  all  relevant  system  disciplines,  and  a  simple  self  assessment  tool  is 
provided  (Figure  3). 

SRL  Levels 
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Note: 

Each  bOK  on  the  matrix  represents  a  Key 
Output  for  that  system  discipline,  The 
colours  represent: 

Gieeii:  full  achievement  of  the  required  outputs 
Amhei:  some  shortfalls  in  the  required  outputs 
Retl:  significant  shortfall  in  the  required  outputs. 


Figure  2.  Outline  of  UK  SRL  Matrix  with  a  Representative  Project  SRL  Signature  Matrix 

{Applying,  n.d.) 

In  the  generic  SRL  matrix,  each  SRL  level  from  1  to  9  is  broken  down  into  key 
outputs  that  need  to  be  achieved  for  each  of  the  systems  disciplines,  including  Systems 
Engineering  Drivers,  Training,  Safety  and  Environment,  Reliability  and  Maintainability, 
Human  Factors  Integration,  Software,  Information  Systems,  Airworthiness,  Project  Specific 
Areas.  For  any  project,  an  SRL  assessment  produces  a  “signature”  rather  than  an  absolute 
SRL  figure.  The  signature  records  the  variation  of  maturity  that  has  been  achieved  across 
the  systems  disciplines,  acknowledging  that  not  all  projects  mature  against  the  systems 
disciplines  at  a  consistent  rate.  The  signature  shown  in  Figure  3  would  be  typical  of  a  project 
in  the  Assessment  phase  (i.e.,  post  Initial  Gate  where  the  target  is  to  achieve  SRL1),  but 
indicates  insufficient  maturity  to  proceed  to  Main  Gate  (the  target  being  SRL4). 


Integration  levels  are  the  next  step.  When  brought  together  with  TRLs  and  SRLs, 
Integration  Readiness  Levels  (IRLs)  provide  a  means  of  progressively  measuring  project 
maturity  at  technology,  component,  sub  system  and  whole  system  levels.  Maturity  is 
measured  in  terms  of  cost,  schedule  and  risk  and  should  also  take  into  consideration  the 
impact  of  the  overall  system  on  the  systems  with  which  it  is  to  operate,  i.e.,  in  a  “system  of 
systems”  context.  IRLs  are  used  at  a  prime  contractors  and  systems  integrator  level,  and 
aim  to  support  successful  transition  of  technologies  to  and  between  acquisition  partners. 
IRLs  are  used  to  support  strategic,  business  and  political  progression,  such  as  funding 
considerations  (national  or  private),  economic  offsets  and  inter-government  partnerships. 


We  can  consider  the  three  (TRL,  SRL,  IRL)  measurements  as  interrelated  but 
independent  sets  of  assessment,  and  use  these  assessments  to  build  up  a  visual  and 
numerical  representation  of  maturity.  Using  the  Red,  Amber,  Green  (RAG)  notation,  this 
representation  resembles  a  wall  of  bricks  (Figure  4)  and  is,  therefore,  termed  a  wall  matrix 
view.  The  lowest,  and  foundation  levels,  are  the  TRLs,  and  the  number  of  layers  relates  to 
the  number  of  key  components.  The  next  tranche  are  the  SRLs  that  cover  the  nine  key 
categories.  The  top,  capping,  level  is  the  IRL,  which  can  also  be  referred  to  as  the  System  of 
System  (SoS)  integration  level.  Any  project,  of  any  complexity,  can  be  evaluated  using  the 
wall  matrix  construct.  As  with  any  assessment,  the  wall  matrix  is  linked  to  a  specific  time  in 
the  life  of  the  project,  and  so  represents  a  snapshot  of  maturity.  Relating  these  “time  now” 
wall  matrix  assessments  to  the  costing  lifecycle  was  the  next  step  of  the  research. 
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Figure  3.  TRL,  SRL,  IRL  Wall  Matrix  Construct 

Costing  Lifecycle 

The  costing  lifecycle  is  a  method  of  project  representation  in  which  all  costs  arising 
from  owning,  operating,  maintaining  and  disposing  of  a  project  are  considered.  Project  costs 
take  into  account  the  initial  cost,  including  capital,  investment  cost,  development  cost,  unit 
purchase  and  installation  cost,  and  future  costs,  including  operating  cost,  energy  cost, 
maintenance  cost,  capital  replacement  cost,  financing  cost  and  any  resale  salvage  for 
disposal  cost  over  the  life  time  of  a  project  or  technology  product.  Any  new  product  or 
project  progresses  through  a  sequence  of  stages  (Figure  5).  Figure  5  expresses  the  profile 
against  an  axis  of  sales,  but  the  axis  could  equally  represent  investment,  effort  or  cost. 

Using  the  traditional  costing  profile,  and  nesting  the  contribution  of  each  of  the 
stages  of  the  UK  Defence  Acquisition  CADMID3  lifecycle,  a  series  of  contributory  peaks  and 
troughs  can  be  mapped.  Using  Valerdi’s  results  alongside  similar  project  management 
costing  models  from  Kerzner  (2003),  a  lifecycle  costing  profile  has  been  constructed  (Figure 
6)  and  then  roughly  aligned  to  the  UK  CADMID  process.  Each  contributory  cycle  has  a 
peak,  which  represents  the  achievement  of  the  aim  of  that  stage  of  the  lifecycle. 

There  are  a  number  of  reports  that  relate  to  early  investment,  resulting  in  lower 
overall  project  cost.  Jorgensen  and  Sjoberg  (2001)  examined  a  portfolio  of  software 
projects,  and  examined  error  occurrences  as  related  to  overrun  costs.  Their  results  show 
early  investment  was  associated  with  early  development  of  maturity,  and,  as  a  result, 
minimum  rework  and  lower  overall  project  cost  (Jorgensen  &  Molokken-Ostovold,  2004; 
Jorgensen  &  Grimstad,  2005;  Jorgensen,  2009).  Valerdi  (2005)  and  Valerdi  et  al.  (2007) 
constructed  a  cost  modeling  tool  that  they  validated  against  industrial  projects.  Using  this 


3  CADMID  means  Concept,  Assessment,  Demonstration,  Manufacture,  In-Service  and  Disposal. 
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evidence  with  the  nested  costing  lifecycle,  it  can  be  concluded  that  contribution  of  funding 
prior  to  the  peak  accelerates  that  stage  of  the  process;  contribution  after  the  peak  ensures 
appropriate  closure  of  that  stage  and  contributes  to  the  start  up  of  the  next  stage. 

The  research  then  focused  on  assessing  when  contributory  peak  was  reached,  in 
relation  to  the  maturity  levels,  with  the  premise  that  investment  prior  to  the  peak  was 
constructive  up  to  the  peak  point,  at  which  point  it  was  better  to  invest  in  the  next  ascending 
stage  of  the  process. 


Figure  4.  Product  Lifecycle  Diagram 

(Gorhcels,  2000) 


Figure  5. 


Lifecycle  Costing  with  Key  Contributing  Stages  Mapped  Against  CADMID 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


691 


Costing  Case  Studies 

Using  historical  cost  and  progress  data,  a  number  of  projects  have  been  examined. 
The  cost  data  has  been  sourced  from  UK  National  Audit  Office  (NAO)  reports  (Stationery 
Office,  2006,  November  24,  p.  14;  2008,  p.  5;  2006,  p.  57;  2007a,  p.  4;  2007b,  p.  6,313; 
2003)  to  minimize  supplier  interpretation,  and  selective  accounting.  The  project  data  has 
also  been  sourced  from  NAO  reports  alongside  other  open  source  publications. 

Unlike  other  studies,  access  to  data  was  not  the  issue;  the  data  was  in  abundance. 
The  real  issue  was  in  consistent  interpretation  from  the  differing  types  of  source  document 
and  in  the  maturity  assessments,  which  were  based  on  technical  reports  and  publications. 
For  the  purposes  of  this  openly  published  paper,  the  case  studies  have  been  deliberately 
censured.  All  the  source  data  is  available  openly,  but  some  has  been  interpreted  in  the 
context  of  information  that  is  privileged  to  the  UK  MoD. 

During  the  period  August  2009  and  March  2010,  twelve  projects  were  researched  for 
source  data.  The  research  related  to  gaining  published  material  that  could  be  assessed 
against  the  costing  and  the  three  maturity  criteria.  The  aim  was  to  provide  source  data  that 
could  be  validated  by  the  authors  and  by  a  technical  judgment  panel  of  engineers  and 
scientist  proficient  with  the  UK  acquisition  lifecycles  and  level  assessments.  The  panel  was 
sourced  from  colleagues  with  Dstl,  who  have  familiarity  with  the  assessment  process  and 
minimal  knowledge  or  involvement  in  the  selected  projects.  The  projects  were  assessed 
against  their  known  timeline,  the  theoretical  costing  lifecycle,  published  costs  and 
associated  panel  TRL,  SRL,  IRL  assessments.  Each  project  was  assessed  at  various  points 
in  its  lifecycle.  The  number  of  assessment  points  varied  from  two  to  twelve.  Larger,  long¬ 
term  projects  were  replete  with  published  data  and  so  were  assessed  at  multiple  points;  for 
some  smaller  projects,  data  was  only  available  at  two  points.  For  each  project,  a  consistent 
selection  of  components  was  assessed  for  TRL.  The  TRLs,  the  nine  aspects  of  SRL  and  the 
capping  IRL  were  then  discussed  using  the  source  data  and  assessed  on  a  wall  matrix  by 
the  panel. 

Once  a  timeline  position  and  an  associated  “RAG”  wall  matrix  had  been  established, 
the  average  TRL,  SRL,  IRL  score  was  calculated  and  plotted  against  the  known  cost  curve 
of  the  project.  Some  plot  points  were  on  an  ascending  line,  e.g.,  prior  to  peak,  and  some 
were  on  the  declining  line.  In  total,  twenty  eight  wall  matrix  plots  were  constructed  and 
associated  with  a  costing  timeline.  Some  of  the  projects  were  reported  on  for  three 
consecutive  years  of  NAO  assessment  so  that  a  robust  element  of  progression  could  be 
established.  After  ratification  and  review,  a  consolidated  plot  was  constructed  (Figure  7). 
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Figure  6. 
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Overview,  Conclusions  and  Further  Work 

After  reviewing  and  consolidating  the  data,  a  table  was  produced  comparing  the 
acquisition  lifecycle  phases  of  CADMID  against  the  set  of  points  of  significance  for  the  three 
readiness  levels,  extracting  for  the  wall  matrix  (Table  2).  From  the  data,  the  optimum 
investment  date  could  not  be  established,  but  a  limit  point  was  found.  Investments  in  a 
particular  stage  added  benefit  to  a  TRL,  SRL,  IRL  limit  point;  investment  after  that  point  was 
required  to  ensure  project  progression,  but  did  not  represent  the  peak  exchange. 

Table  2.  Consolidated  Wall  Matrix  and  Timeline  Table  of  Results 


TRL  point  of 
significance 

SRL  point  of 
significance 

IRL  point  of 
significance 

CONCEPT 

<3.2 

<2.4 

N/A 

ASSESSMENT 

<  5.6 

<  6.5 

<  1.8 

DEMONSTRATION 

<  7.6 

<  7.8 

<4.4 

MANUFACTURE 

<8.1 

<8.4 

<8.2 

IN-SERVICE 

N/A 

N/A 

<9.0 

There  was  a  large  variation  in  the  sets  of  TRL,  SRL  and  IRL  data  that  will  be 
examined  in  further  studies.  The  source  of  variation  could  be  related  to  source  data  error, 
interpretation  or  factors  that  made  the  specific  project  different  to  its  peer  group,  e.g., 
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groundbreaking  atomic  physics  as  opposed  to  next  generation  vehicle  development. 
Expanding  the  analysis  to  a  wider  pool  to  projects  would  refine  the  technique  and  increase 
the  robustness  of  the  “points  of  significance”  as  well  as  reduce  the  statistical  effects.  The 
use  of  one  technical  judgment  panel  was  convenient  for  this  stage  of  the  research  but  could 
introduce  experimental  errors  over  time.  Familiarity  and  prior  knowledge  of  previous  results 
started  to  bias  conversations  on  some  of  the  latter  assessments  of  larger  projects. 

Looking  at  the  global  and  national  economic  situation,  the  competition  for 
government  funding  is  becoming  more  intense,  and  departments  such  as  the  MoD  are 
under  increasing  pressure  to  deliver  optimum  results  against  investment.  This  investment  is 
not  just  what  to  invest  in,  but  also  when  to  invest.  The  extant  UK  MoD  assessment  criteria 
include  TRL,  SRL  and  IRL  decision  points,  so  this  data  is  already  available.  By  mapping  the 
optimum  investment  point  against  the  TRL,  SRL,  and  IRL,  a  wall  matrix  could  guide  the 
diversion  of  funds  into  projects  up  to  key  points,  which  would  result  overall  in  maximizing  the 
outcome  benefit  to  UK  Defence. 
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2003  -2010  Sponsored  Research  Topics 


Acquisition  Management 


Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

BCA:  Contractor  vs.  Organic  Growth 
Defense  Industry  Consolidation 
EU-US  Defense  Industrial  Relationships 

Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to  Shipyard 
Planning  Processes 

Managing  the  Services  Supply  Chain 

MOSA  Contracting  Implications 

Portfolio  Optimization  via  KVA  +  RO 

Private  Military  Sector 

Software  Requirements  for  OA 

Spiral  Development 

Strategy  for  Defense  Acquisition  Research 

The  Software,  Hardware  Asset  Reuse  Enterprise  (SHARE)  repository 


Contract  Management 


Commodity  Sourcing  Strategies 
Contracting  Government  Procurement  Functions 
Contractors  in  21st-century  Combat  Zone 
Joint  Contingency  Contracting 

Model  for  Optimizing  Contingency  Contracting,  Planning  and  Execution 

Navy  Contract  Writing  Guide 

Past  Performance  in  Source  Selection 

Strategic  Contingency  Contracting 

Transforming  DoD  Contract  Closeout 

USAF  Energy  Savings  Performance  Contracts 

USAF  IT  Commodity  Council 

USMC  Contingency  Contracting 


Financial 


Management 


Acquisitions  via  Leasing:  MPS  case 


Budget  Scoring 


Budgeting  for  Capabilities-based  Planning 
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Capital  Budgeting  for  the  DoD 

Energy  Saving  Contracts/DoD  Mobile  Assets 

Financing  DoD  Budget  via  PPPs 

Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition  Budgeting 
Reform 

PPPs  and  Government  Financing 
ROI  of  Information  Warfare  Systems 
Special  Termination  Liability  in  MDAPs 
Strategic  Sourcing 

Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 


Human  Resources 


Indefinite  Reenlistment 

Individual  Augmentation 

Learning  Management  Systems 

Moral  Conduct  Waivers  and  First-tern  Attrition 

Retention 

The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 
Tuition  Assistance 


Logistics  Management 


Analysis  of  LAV  Depot  Maintenance 
Army  LOG  MOD 
ASDS  Product  Support  Analysis 
Cold-chain  Logistics 

Contractors  Supporting  Military  Operations 
Diffusion/Variability  on  Vendor  Performance  Evaluation 
Evolutionary  Acquisition 

Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 
Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

Optimizing  CIWS  Lifecycle  Support  (LCS) 

Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance  Activity 
Pallet  Management  System 
PBL  (4) 

Privatization-NOSL/NAWCI 
RFID  (6) 
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■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module  Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our  website: 
www.acquisitionresearch.org 
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Why  does  DSTL  exist? 


Lead  and  support  the  defence  science 
and  technology  (S&T)  community  in 
responding  to  MOD’S  current  and  future 
needs. 

Act  as  MOD’S  S&T  agent  and  provides  the 
single  S&T  focus. 

Work  with  industry  and  academia  to 
increase  the  S&T  impact. 

Work  with  Other  Government 
Departments  to  exploit  our  expertise  and 
knowledge. 

Enhance  the  safety  and  security  of  UK 
citizens  and  interests. 

Provide  independent  and  impartial  advice. 

Accountable  to  Government  /  taxpayer  / 
front  line  troops 


Terrain  following  in 
the  vicinity  of 
Snowdonia. 


The  Pentagon,  ■  •  i  ■ 

in  Washington 


_ __ 


Working 

conditions  in  the 
High  Mojave 
Desert,  at  Fort 
Irwin,  California 


Maximising  the  impact  of  science  and  technology  for  the  defence  and  security  of  the  UK 
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The  Acquisition  Pot 


•  UK  MoD  spends  £2.6Bn/year  on 
research  and  development 

•  £500m  on  laboratory  research 

•UK  NAO  shows  -50%  project  overruns 
are  due  to  late  technology  maturation 

•  US  evidence  shows  technology  matured 
prior  to  Main  Gate  results  in  small  time 
and  cost  overruns 

•  Paper  examines  funding  profiles  as 
aligned  to  maturity  levels  of  technology, 
system  and  integration  readiness 


Funding  for  Life:  When  to  Spend  the  Acquisition  Pot? 
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When  is  the  best  time  to  spend  -  globally? 


•  Global  economic  growth  (trend  rate  of  3-4%)  has  been  driven 
by  globalisation  over  the  last  30  or  more  years  1  We  now 
operate  in  a  multi-faceted  economic  landscape  that  is 
intimately  interconnected  and  influential. 

•  In  Sept  2009  ,  the  UK  Chancellor  of  the  Exchequer  forecast 
that  in  April  2010  public  borrowing  would  exceed  12%  of  GDP  2 

•  UK  Spending  plans  from  201 1  will  be  “very  tough  -  the 
toughest  for  decades”  but  refused  to  outline  how  this  will  affect 
public  services,  except  to  say  that  the  Government  would 
“protect  spending  on  those  frontline  public  services  on  which 
we  all  depend”.  ' 

•  In  the  US,  the  current  focus  has  been  on  the  US  Health 
systems  and  associated  reforms. 

•  In  the  UK,  balancing  the  books  and  the  impending  election 
process  has  raised  the  level  of  scrutiny  and  resulted  in  long 
term  plans  being  put  on  hold  until  the  political  direction  and 
stability  have  been  secured  3 


1.  International  Monetary  Fund  (IMF)  (2008),  “World  Economic  Outlook  (WEO)  2008:  Financial  Stress,  Downturns,  and  Recoveries”,  October  2008,  pp  3  -  44.  Retrieved  October  16, 
2008  from  http://www.imf.org/external/pubs/ft/weo/2008/02/pdf/text.pdf 

2.  The  Economist,  “Britain's  sharp  budget  choices”,  published  on  Sep  24th  2009,  From  The  Economist  print  edition  http://www.economist.com 

3.  Ravlin,  A.M.  and  Swahili,  I.V.  (2004),  “Restoring  fiscal  sanity:  how  to  balance  the  budget”,  Library  of  Congress  Cataloging-in-Publication,  The  Brookings  Institution,  Washington  DC. 
Chapter  6  “Meeting  the  Revenue  Challenge”  by  Henry  J  Arron,  William  G.  Gale  and  Peter  R  Orszag,  Pg  1 1 1  - 127  www.brookings.edu  ISBN  0-8157-7781-7 


“balancing  the  books” 
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When  is  the  best  time  to  spend  -  nationally? 


•  Global  economic  context  places  more  pressure 
on  all  dimensions  of  UK  government  funding, 

•  UK  Defence  spending  is  £40bn,  which 
represents  5.6%  of  the  £704bn  total 1 

•  Defence  planning  is  a  long  term  activity 

•  Success  related  to  the  stability  of  the  aims  and 
contributory  components 

•  In  the  current  global  and  national  climate,  it  is 

not  a  good  time  to  spend,  however . 

•  ...  to  maintain  capability  in  defence  and  other 
sectors,  spending  has  to  be  sustained  to  ensure 
stability  and  maintenance;  some  projects  are 
committed  by  still  under  development  and  some 
projects  are  further  down  the  in-service  and 
support  stage 


1  The  Economist,  “HM  Treasury  2010-1 1  Near  Cash  Projections”  published  on  Mar  18th  2010,  From  The  Economist  print  edition  http://www.economist.com 


Defence  spending  has  to  be  sustained  to  ensure  stability  and  maintenance 
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Technology,  System  and  Integration  Investment 


Examine  costing  profile  as  related  to  three 
system  measures  of  readiness: 

-  Technology  Readiness  Levels  (TRLs) 

•  can  be  used  to  determine  the 
current  component  maturity 

-  System  Readiness  Levels  (SRLs) 

•  project  management  tool  to 
capture  evidence,  and  assess  and 
communicate  system  maturity 

-  Integration  Readiness  Levels  (IRLs) 

•  System  of  System  (SoS) 
integration  level 

Measurements  are  interrelated  but 
independent  sets  of  assessment 12 


National  Aeronautics  and  Space  Administration  (NASA)  Techi 
Readiness  Level 

1.  Basic  principles  observed  and  reported 

2.  Technology  concept  and/or  application  formulated 

3.  Analytical  and  experimental  critical  function  and/or  characte 
proof  of  concept 

4.  Component  and/or  breadboard  validation  in  laboratory 
environment 

5.  Component  and/or  breadboard  validation  in  relevant  enviroi 

6.  System/subsystem  model  or  prototype  demonstration  in  a 
relevant  environment  (ground  or  space) 

7.  System  prototype  demonstration  in  a  space  environment 


8.  Actual  system  completed  and  'flight  qualified'  through  test  a 
demonstration  (ground  or  space) 

9.  Actual  system  'flight  proven’  through  successful  mission 
operations 


System  Test,  Launch 
&  Operations 


System/  Subsystem 
Development 


Technology 

Demonstration 


Technology 

Development 


Research  to  Prove 
Feasibility 


Basic  Technology 
Research 


TRL  A 

TRL  3 
TRL  2 
TRL  1 


1  “Systems  Engineering  Leading  Indicators  -  Enabling  Assessment  of  Acquisition  Technical  Performance ”,  Paul  Montgomery  and  Ron  Carlson,  Naval  Postgraduate  School 

2  “System  Capability  Satisficing  in  Defense  Acquisition  via  Element  Importance  Measures”  Brian  Sauser,  Jose  Ramirez- Marquez,  David  Nowicki,  Weipiong  Tan,  Romulo 
Magnaye,  Matin  Sarfaraz,  Stevens  Institute  of  Technology;  and  Abhi  Deshmukh,  Texas  A&M  University 


Project  maturity  at  technology,  component,  sub-system  and  whole  system  levels 
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System  Readiness  Level  (SRL)  matrix 


Projects  track  their  maturity  through 
the  nine  SRL  steps 

Across  all  relevant  system 
disciplines 

Red,  Amber,  Green  (RAG) 
notation. 

Simple  self  assessment  tool 

Produces  a  ‘signature’  rather  than 
an  absolute  SRL  figure. 

Linked  to  a  specific  time  in  the  life 
of  the  project,  and  so  represents  a 
snapshot  of  maturity. 


3RL  Levels 

3  4  5  6  7  3  0 
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Hole: 

Each  box  on  the  matrix  represents  a  Key 
Output  for  that  system  discipline.  The 
colours  represent: 

Gieen:  full  achievement  of  the  required  outputs 
Amber:  some  shortfalls  in  the  required  outputs 
Reil:  significant  shortfall  in  the  required  outputs, 


Presents  a  snapshot  of  project  maturity  based  on  system  readiness  levels 
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Wall  Matrix  of  Readiness  Levels 


Wall  matrix  view  of  TRL,  SRL, 
and  IRL 

Any  project,  of  any  complexity 
can  be  evaluated  using  the 
wall  matrix  construct. 

Need  to  assess  “time  now“ 
wall  matrix  assessments  as 
related  to  the  costing  lifecycle 
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Presents  a  snapshot  of  project  maturity  based  on  a  combination  of  readiness  levels 
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Costing  Life  Cycle 


•  All  costs  arising  from 
owning,  operating, 
maintaining  and  disposing 
of  are  considered  1 

•  Nested  costing  lifecycle 

•  Profiled  against  UK 
CADMID2  acquisition 
lifecycle  stages 

•  Funding  contribution 
timing  with  relation  to  the 
peak  affects  stage 
success  outcome 


1  Gorhcels,  L.  (2000)  “The  Product  Manager’s  Handbook:  The  Complete  e  Management  Resource,  2nd  Edition,  McGraw-Hill,  London. 

2  CADMID  -  Concept,  Assessment,  Demonstration,  Manufacture,  In-Service  and  Disposal. 

Method  of  project  representation,  in  which  all  costs  arising  from  a  project  are  considered 
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Costing  Case  Studies 


•  August  2009  and  March  2010, 
twelve  projects  were 
researched  for  source  data 

•  The  aim  was  to  provide  source 
data  that  could  be  validated  by 
the  authors,  and  by  a  technical 
judgment  panel  of  engineers 
and  scientist  proficient  with  the 
UK  acquisition  lifecycles  and 
level  assessments 

•  The  projects  were  assessed 
against  their  known  timeline, 
the  theoretical  costing 
lifecycle,  published  costs  and 
associated  panel  TRL,  SRL, 
IRL  assessments 

•  Each  project  was  assessed  at 
various  points  in  its  lifecycle 


12  projects  were  assessed  against  the  costing  and  the  three  maturity  criteria. 
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Overview  and  Conclusions 


•  Systems  engineers  often  serve  as 
technical  points  of  contact  throughout 

the  entire  system  lifecycle  -  they  are 
the  single  point  of  “project  truth” 

•  “Limit  point"  was  established  where 
investments  in  a  particular  stage  added 
benefit  to  a  TRL,  SRL,  IRL 

•  “Optimum  investment  date”  could  not 
be  established  using  the  limited 
number  of  studies  -  more  to  follow  ... 


•  Conclusion  -  mapping  the  optimum  investment  point  against  the  TRL, 
SRL,  IRL  wall  matrix  could  guide  the  diversion  of  funds  into  projects  up 
to  key  points. 


Individual  CADMID  stage  investment  limit  points  identified  -  but  not  “optimised” 


TRL  point  of 
significance 

SRL  point  of 
significance 

IRL  point  of 
significance 

CONCEPT 

<3,2 

<2,4 

N/A 

ASSESSMENT 

<5,6 

<6,5 

<1.8 

DEMONSTRATION 

<7,6 

<7.8 

<4.4 

MANUFACTURE 

<8,1 

<8,4 

<8.2 

IN-SERVICE 

N/A 

N/A 

<9.0 
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dstl  Questions? 


Panel  #22  -  Funding  for  Life:  When  to  Spend  the 
Acquisition  Pot 

Kate  Gill 

Group  Leader  -  Maritime  Systems  Engineering,  DSTL  (kmgill@dstl.gov.uk) 

Dr  Kirsty  Carter-Brown 

Associate  Fellow  -  Maritime  Systems  Engineering,  DSTL  (kcbrown@dstl.gov.uk) 
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